Repository navigation
Conversation
Enable rsync to read directly from a local named pipe (FIFO) or unseekable stream and perform full rolling-checksum delta transfers against an existing destination basis file without staging to temporary storage or changing the wire protocol. Key changes: - Add --stream-source, --stream-max-size=SIZE, --stream-timeout=SECONDS, and --stream-pipe-sz=SIZE CLI options to options.c, usage.c, and rsync.1.md. - Disguise the local FIFO source as S_IFREG with st_size = stream_max_size in make_file() (flist.c) so the receiver/generator negotiates normal basis checksums (s->sums) without protocol modifications. - Implement sequential stream windowing in map_file() and map_ptr() (fileio.c): disallow backward seeks (read_start != map->p_fd_offset), multiplex reads with select() to enforce stream_timeout, clamp map->file_size dynamically on early EOF (n == 0), and optionally resize the kernel pipe ring buffer via F_SETPIPE_SZ on Linux. - Update match_sums() and hash_search() (match.c) to pre-read stream data when s->count == 0 (missing/empty basis file) and safely clamp the sliding window and tail flush upon early EOF. - Verify true EOF in send_files() (sender.c) via a non-blocking read probe when a stream reaches stream_max_size, aborting with IOERR_GENERAL on overflow, and reconcile F_LENGTH(file) and stats.total_size to the actual bytes read before progress and transfer logging. - Ensure receive_data() (receiver.c) calls do_ftruncate(fd, offset) in --inplace mode when an early-EOF stream ends shorter than the basis file. - Block incompatible options (-r, -a, -d, -c, --append, --files-from, --remove-source-files, batch mode) and file-size-dependent options (--size-only, --max-size, --min-size, --fuzzy, --preallocate), while enforcing ignore_times = 1 and clearing update_only, preserve_links, preserve_specials, and preserve_devices. - Add Python test suites (stream-source_test.py and stream-early-eof_test.py) covering delta matching, --inplace truncation, early EOF, exact ceiling, ceiling overflow, idle timeout, and option validation.
|
I think XNU/NetBSD would require extra/different pipe handling for EOF. Will look into that. |
…ld warning On macOS (XNU) and NetBSD, named FIFOs clear their EOF socket state (SBS_CANTRCVMORE) after a 0-byte read() is consumed so the FIFO can be reused by future writers. If map_ptr() is called again before the caller clamps its length to the shrunk buf->file_size, a subsequent select() on the FIFO sees zero writers and no EOF flag, blocking until stream_timeout expires. Track stream EOF via eof_reached in fileio.c (reset in map_file()) so map_ptr() and send_files() never invoke select() or read() on the stream descriptor once a 0-byte read() has been observed. Also guard buf before dereferencing buf->file_size in match_sums() to silence a Clang scan-build null-dereference warning.
0205d26 to
6504328
Compare
| struct timeval tv; | ||
|
|
||
| FD_ZERO(&rfds); | ||
| FD_SET(map->fd, &rfds); |
There was a problem hiding this comment.
FD_SET is undefined when the descriptor reaches FD_SETSIZE -- With 1100 =< inherited descriptors there can be out-of-bounds access under UBSan and a stack-buffer-overflow under ASan. Use poll() or ppoll() and add the high-descriptor regression.
- Verify EOF or excess stream bytes in match_sums() before finalising the transfer checksum so streams exceeding --stream-max-size fail with RERR_FILEIO (exit code 11) without truncating or overwriting existing destination files, and stalled streams at the size boundary honour --stream-timeout via poll(). - Track --stream-timeout explicitly with OPT_STREAM_TIMEOUT instead of a -1 sentinel so negative values disable the timeout consistently with 0, matching the documented behaviour. - Reject multiple source arguments in start_client() before opening any FIFO when --stream-source is active. - Update rsync.1.md to document pre-finalisation abort behaviour on oversized streams and the single-source requirement. - Expand stream-source_test.py to cover exact-ceiling transfers, oversized stream aborts preserving destination files, paused/stalled stream timeouts, and writerless multi-FIFO rejection in both argument orders.
|
I think XNU should have different handling with poll also. |
|
@steadytao poll would introduce inconsistency and weird behavior with XNU due to how it handles EOF. I think I might fall back to |
|
@steadytao Also, XNU supports expanding FD_SETSIZE (Linux do not) at compile time via _DARWIN_UNLIMITED_SELECT, I'll bump FD_SETSIZE in fileio.c alongside the select() fallback and bounds check so high file descriptors still work cleanly on macOS without dynamic allocations or complex handling. |
- Fall back to select() under __APPLE__ in wait_stream_fd() because XNU's poll() fails to wake on FIFO EOF when the pipe buffer is empty, causing 0-byte and exact-ceiling --stream-source transfers to time out on macOS. - Enable _DARWIN_UNLIMITED_SELECT and expand FD_SETSIZE to 65536 before including headers in fileio.c alongside an explicit bounds check so high inherited file descriptors remain supported.
steadytao
left a comment
There was a problem hiding this comment.
Needs rebasing + testsuite consolidation. Please port the stream tests to the current harness and cover local copies, remote-shell and daemon pushes, interoperability with an older receiver and both normal and --inplace writes.
Also. The PR description still says an oversized stream is truncated successfully. Please update it to say that exceeding --stream-max-size fails the transfer.
| full_fname(thisname)); | ||
| return NULL; | ||
| } | ||
| if (is_stream() && S_ISFIFO(st.st_mode)) { |
There was a problem hiding this comment.
With --stream-source --links a symlink to a FIFO is copied as a symlink and rsync exits successfully without reading the stream. Please reject the source before adding it to the file list unless it is a FIFO after applying the requested symlink handling.
| if (is_stream()) { | ||
| if (!S_ISFIFO(st.st_mode)) { |
There was a problem hiding this comment.
The blocking open above runs before the stream timeout starts. A FIFO with no writer still blocks with --stream-timeout=1; please make the timeout cover the initial wait for a writer and add a regression test.
This PR introduces
--stream-source, enablingrsyncto perform full rolling-checksum delta transfers directly from non-seekable named pipes (FIFOs) and process substitutions (when combined with-L/--copy-links) into local or remote regular destination files.Previously, passing a FIFO to
rsynceither copied the special FIFO device node itself (--specials) or skipped it entirely.--stream-sourcedisguises the source FIFO as a regular file inside the file list, reads the incoming stream sequentially throughmap_ptr(), computes rolling block/file checksums against the receiver's basis file on the fly, and seamlessly truncates the logical file length if the stream reaches EOF before the configured ceiling.Key Features & Options
--stream-source: Instructs the local sender to treat a single local source FIFO as a regular file and stream its contents through the delta-transfer engine.--stream-max-size=SIZE(Required): Sets the initial virtual file size ceiling advertised in the file list before the stream is consumed.--stream-timeout=SECONDS: Configures aselect()read timeout on the FIFO descriptor insidemap_ptr()(default:30seconds;<= 0disables the timeout, matching--timeoutbehavior).--stream-pipe-sz=SIZE: Adjusts the kernel pipe ring-buffer capacity on Linux viafcntl(fd, F_SETPIPE_SZ, ...)within[64K, 16M](default:1M) to reduce context-switch overhead between the writer process andrsync.Why
--stream-max-sizeis Required as an Initial CeilingNamed pipes report
st_size == 0viastat(), and the true byte length of a live stream cannot be known until the writer closes the pipe. However,rsync's protocol and delta-transfer engine require a file size during theflistphase (make_file()) and before enteringmatch_sums()so that:rsyncpeers).s->sums) covering the destination file.map_file()andhash_search()can initialize the sliding window bounds (lenandend).--stream-max-sizeacts strictly as an upper-bound ceiling, not an exact size requirement:--stream-max-size): Users can safely pass a generous ceiling (e.g.,--stream-max-size=10Gfor a database dump that turns out to be42Mor even0bytes). Whenread()inmap_ptr()hits0(EOF),map->file_sizeandmap->p_lenshrink to the exact byte count read (map->p_fd_offset). Bothhash_search()andmatch_sums()detectbuf->file_size < len, clamplendynamically, and flush the remaining literals/tokens up to the real EOF so the reconstructed destination file (both temporary-file and--inplacemodes) has the exact stream byte length. On the sender side,sender.creconcilesstats.total_sizeandstats.total_transferred_sizeto the actual bytes read. Because the initialflistentry transmitsstream_max_sizewithout a wire protocol change, the advertised ceiling still controls%lformat logging, transfer progress percentages, and receiver-side (or daemon-side) total-size statistics.--stream-max-size): If a producer writes more bytes than--stream-max-size,rsyncstops reading at the exact limit, probes the pipe with a 1-byte read insender.cto detect unconsumed data, logs a warning, and completes the transfer cleanly at the ceiling size without overflowing.Assigning Virtual File Metadata
Because
make_file()converts the FIFO's mode fromS_IFIFOtoS_IFREGwhile preserving the rest of thestatstructure, standardrsyncmetadata options work normally on the resulting destination file:--perms(-p) and--chmod(e.g.,--perms --chmod=F600to lock down a streamed backup).--owner(-o),--group(-g), and--chown=USER:GROUP.--times(-t), applying the FIFO's mtime or custom modification times to the target regular file.-L/--copy-linksallows streaming from symlinked pipes such as/dev/stdinor shell process substitutions (<(cmd)), using the target file's name on the destination.Option Restrictions & Automatic Overrides
Because a FIFO can only be read once sequentially and its true size is unknown until EOF,
--stream-sourceis restricted to a single local sender stream and blocks options that require directory traversal, multiple passes, or a stable pre-known source size:--stream-sourceis only accepted on the local client acting as the sender (!am_server && !am_daemon && am_sender). Pushing to a local path, remote shell (HOST:DEST), or remote rsync daemon (HOST::DEST/rsync://) works normally without requiring--stream-sourcesupport on the remote peer.-r,-a,-d,--files-from): Stream mode operates on a single explicit FIFO source. While-ais rejected because it implies-rand--specials, individual metadata flags (-p,-t,-o,-g,-L) can still be passed normally.-c/--checksum,--append,--append-verify, batch mode):--checksumwould consume the pipe duringflistgeneration before the transfer starts, and--append/ batch modes conflict with single-pass stream semantics.--size-only,--max-size,--min-size,-y/--fuzzy,--preallocate): Blocked explicitly because the generator only sees the transmittedstream_max_sizeceiling in theflistrather than the final stream length.--remove-source-files: Prevents unlinking persistent named pipes or/dev/fd/*descriptors after transfer.Additionally,
--stream-sourceautomatically forces--ignore-times(ignore_times = 1) and clearsupdate_only,preserve_links,preserve_specials, andpreserve_devicesso the generator never skips the transfer based on quick checks or copies the FIFO node itself instead of its data.Implementation Details
flist.c(make_file): Validates that the source isS_ISFIFO(st.st_mode)whenis_stream()is active, rewritesst.st_modefromS_IFIFOtoS_IFREG, and setsst.st_size = stream_max_size.fileio.c(map_file,map_ptr):fcntl(fd, F_SETPIPE_SZ, stream_pipe_sz)inmap_file()under#ifdef F_SETPIPE_SZ.read_start != map->p_fd_offset) on streams inmap_ptr().select()whenstream_timeout > 0, retries onEINTR, and handles early EOF (nread == 0) by truncatingmap->file_size = map->p_fd_offset, settingmap->p_len = read_offset, and zeroing the unused buffer tail.match.c(hash_search,match_sums):len,end, andkwheneverstream_early_EOF(buf, len)triggers across the initial read, the hash-collision skip path, and the rolling-window advance path.offset - backup(last_match) tomap_ptr()during hash-collision skips (matching the normal byte-by-byte path) so unsent literal bytes (last_match .. offset) stay inmap_ptr()'s sliding buffer instead of being evicted and triggering a backwardlseek()whenmatched()flushes them.match_sums()whens->count == 0so early EOF shrinkslenbeforematched()flushes literal chunks, and guardsmap_ptr(buf, len-1, 1)atfinish_search:withif (len > 0)to handle 0-byte streams safely.sender.c(send_files): Re-verifiesS_ISFIFO, checks whether data remains in the pipe after reachingstream_max_size, and reconciles sender-sidestats.total_sizeandstats.total_transferred_sizewithmbuf->file_size.rsync.1.mdandusage.c, and adds two comprehensive test suites (testsuite/stream-source_test.pyandtestsuite/stream-early-eof_test.py).Feature Proposal: Native Stream with UNIX Pipe discussion (--stream-source) #1117